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Route  1,  Box  619 
Crozet,  VA   22932 
May  21,  1988 


The  Education  Subcommittee  of  the 

Joint  Legislative  Commission  on 

Government  Operations 
North  Carolina  General  Assembly 
State  Legislative  Building 
Raleigh,  NC  27611 

Gentlemen: 

It  is  a  pleasure  to  submit  this  report  of  our  panel 
activities,  findings,  and  recommendations  regarding  the 
performance  appraisal  instriiments .   We  trust  that  we  have 
responded  to  your  charge  appropriately.   We  hope  that  our 
findings  and  recommendations  will  prove  useful  in  the 
months  ahead. 

We  very  much  appreciate  the  courtesies  extended  to  us 
in  the  conduct  of  our  study  by  you,  the  members  of  the 
Subcommittee;  by  Joan  Rose,  contract  administrator;  by  the 
several  members  of  the  Department  of  Public  Instruction 
staff  with  whom  we  met;  and  by  local  school  district 
personnel  during  my  prepanel  visits.   Without  splendid 
cooperation  and  assistance  from  all  these  fine  people,  our 
task  would  have  been  much  more  difficult. 


Sincerely  yours, 
Richard  M.  Brandt 
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EXECUTIVE  SUMMARY 

Our  study  and  review  of  the  TPAI  leads  us  to  one 
overriding  conclusion.   It  is  a  quality  instrument,  one 
that  is  highly  suited  to  its  purposes.   The 
recommendations  we  list  below  represent  suggestions  for 
improving  it  still  further,  especially  to  enhance  its 
effectiveness  as  a  basic  tool  for  making  career  ladder 
decisions  and  for  improving  teaching.   In  no  way  do  these 
recommendations  imply  fundamental  deficiencies  that  would 
call  for  its  elimination  as  the  state  teacher  appraisal 
instrument.   We  consider  our  recommendations  suggestions 
for  fine-tuning  an  already  good  system  of  performance 
appraisal. 

Likewise,  our  limited  commentary  on  the  PPAI  and  other 
specialty  measures  does  not  suggest  they  are  inadequate, 
only  that  we  had  less  to  examine  because  of  their  recent 
development  and  lack  of  widescale  use.   On  the  surface, 
they  look  promising  and,  with  the  few  exceptions  we  have 
noted,  consistent  with  the  TPAI  in  purpose  and  procedure. 

We  summarize  our  recommendations  in  the  order  in  which 
they  are  presented  in  the  text. 
CRITERIA 

1.   Rearrange  some  functions  and  teaching  practices  to  a 
total  of  twelve,  ten  of  which  relate  directly  to 
teaching  and  are  observable  in  the  classroom. 
Specifically; 
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o    Reverse  the  order  of  functions  6  and  7. 

o    Rename  new  function  6  "Communication  and 

Interaction  in  the  Classroom."   Include  function 
6  as  one  of  those  to  be  observed  in  the 
classroom,  and  list  relevant  teaching  practices 
as  follows: 

6.1   Teacher  speaks  fluently  and  precisely 
(currently  3.3) 

^'^  IrtZ^^   presents  the  lesson  or  instructional 
activity  using  concepts  and  language 
understandable  to  the  students  (currently 

^''   Tcu^^i'"^^''^^^^  effectively  with  students 
(currently  part  of  7.2) 

6.4   Teacher  treats  all  students  in  a  fair  and 
eguitable  manner  (currently  7.1) 

Divide  current  function  3  (Instructional 
Presentation)  into  five  specific  functions  as 
follows: 

o    Initiating  Instruction 

o    Motivating  Students 

o    Managing  Routines  and  Transitions 

o    Presenting  Accurate  and  Appropriate  Content 

o    Providing  Closure 

Teaching  practices  for  these  latter  functions 

would  be  taken  from  current  function  3  (excluding 

3.3  and  3.4)  and  from  such  instruments  as  the 

Connecticut  Competency  Instrument. 

Expand  the  literature  search  for  effective 
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teaching  practices  to  include  coaching,  modeling, 

cooperative  and  mastery  learning  models  of 

teaching  in  addition  to  process-product  research 

on  direct  instruction 

o    Encourage  local  districts  to  identify  practices 

that  reflect  local  needs  for  the  two  functions 

that  are  less  observable  in  the  classroom  and 

relate  less  directly  to  instruction,  i.e.: 

o    Facilitating  Instruction  (currently  F6) 

o    Performing  Non-Instructional  Duties 
(currently  F8) 

FODI  AND  POST-OBSERVATION  CONFERENCE 

2.  In  addition  to  scripting,  consider  collecting  other 
kinds  of  observational  data  on  a  pilot  basis  such  as 
student  participation  and  on-task  behavior,  teaching 
patterns,  and  other  classroom  action.   While  we  do  not 
recommend  changing  scripting  procedures  at  this  time, 
other  kinds  of  data  might  well  be  needed  in  the  future 
to  assist  in  strengthening  the  objectivity  and  power 
of  the  TPAI  to  differentiate  between  good  and  superior 
teachers. 

FODI 

3.  Provide  teachers  with  copies  of  all  descriptive  data 
(FODIs  and  any  other  descriptive  instruments  used)  in 
advance  of  the  post-observation  conference. 
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FODA 

4.  Provide  space  on  the  FODA  form  for  the  evaluatee  to 
respond  to  observations  and  quality  assessments  made 
by  the  evaluator. 

TRAINING 

5.  Provide  evaluators  with  specific  training  in 
conferencing  skills  and  using  the  FODI  and  FODA  in 
post-observation  conferences. 

SCALE  STEPS 

6.  Maintain  the  six-point  rating  scale. 
RATING  PROCEDURE 

7.  Follow  a  step-by-step  procedure  in  making  summative 
judgments: 

a.  Is  performance  at  least  at  standard? 

b.  If  so,  is  it  at  or  somewhere  above  standard? 

c.  If  above,  is  it  merely  above  standard  (4)  or  well 
above  (5)  or  superior  (6)? 

PDF 

3.   Revise  the  Professional  Development  Plan  form  somewhat 
to  reflect  more  highly  individualized,  professional 
development  beyond  basic  skill  competency,  especially 
for  above  standard  teachers. 

PORTFOLIO 

9.  Develop  a  form  and  procedure  for  recording  and 
analyzing  portfolio  contents  systematically  in 
connection  with  the  non-classroom  functions.   Such 


forms  and  procedures  should  be  tailored,  however,  to 
local  district  priorities. 
INTERVIEW 

10.  Develop  a  structured  interview  format  for  collecting 
data  from  principals,  special  groups  (e.g.  school 
psychologists,  counselors,  etc.)  and  teachers  in  a 
systematic,  consistent  fashion  about  functions  not 
readily  observed  in  the  classroom. 

CERTIFICATION  OF  EVALUATORS 

11.  Require  all  evaluators  to  be  certified  for  this  role 
by  passing  a  performance-based  test  of  their 
competency  to  use  the  TPAI  accurately  and 
consistently.   Evaluators  should  be  required  to 
demonstrate  their  competence  on  a  regular  basis 
against  criterion  measures.   They  should  not  be 
permitted  to  function  as  evaluators  unless  they  are 
certified  and  remain  certified  as  evaluators. 

CAREER  LADDER  DECISIONS 

12.  Restrict  the  decision-making  use  of  the  TPAI  to 
certification  and  career  levels  I  and  II  decisions. 
Do  not  attempt  to  use  it  as  the  primary  measure  for 
career  level  III  decisions. 

SUMMATIVE  RATINGS  BY  CONSENSUS 

13.  Require  the  joint  participation  of  OEs,  principals  and 
other  evaluators  who  are  involved  in  the  data 
collection  process  in  the  final  summative  judgment. 


Training  in  achieving  data-based  consensus  is  also 
recommended. 
CONSECUTIVE  OBSERVATIONS 

14.  Experiment  with  observing  teachers  up  to  four 
consecutive  days  teaching  the  same  class  for  possible 
inclusion  of  such  a  requirement  for  career  level  II 
applicants. 

JOINT  OBSERVATIONS/ANALYSES 

15.  Create  panels  of  administrators  and  teachers  to  review 
and  rate  samples  of  FODIs  and  FODAs  for  comparability. 
Conduct  joint  observations,  gather  data,  and  share 
analyses  to  enhance  the  consistency  of  instrument  use 
and  interpretation. 

RELIABILITY  STUDIES 

15.  Conduct  various  studies  of  rating  reliability  at  both 
the  state  and  district  levels  to  assess  the 
consistency  across  raters  and  the  stability  of  teacher 
patterns  from  one  time  to  another. 

VALIDITY  STUDIES 

17.  Conduct  validity  studies  within  districts  of  (a) 

achievement  gain  scores  of  students  taught  by  teachers 
rated  at  standard  (#3)  versus  those  taught  by  teachers 
rated  5  and  6;  and  (b)  student  gains  in  pilot 
districts  versus  those  in  comparable  non-pilot 
districts. 
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UNIVERSITY  RESEARCH  ASSISTANCE 

18.  Request  assistance  from  university  faculty  and  provide 
graduate  student  financial  support  for  the  conduct  of 
TPAI  reliability  and  validity  studies  in  the  pilot 
effort. 

WORD  PROCESSORS 

19.  Provide  evaluators  with  word  processors  and 
appropriate  training  as  needed  to  assist  them  in 
writing  FODIs  and  FODAs. 

INCREASE  OF  OEs 

20.  Reduce  the  ratio  of  teachers  to  OEs  to  48  to  1  in 
order  to  increase  the  amount  of  assistance  available 
to  teachers  for  remediating  deficiencies  and  improving 
their  instruction. 

GROWTH  BEYOND  COMPETENCY 

21.  Focus  career  level  III  criteria  on  leadership 
functions  and  growth  beyond  competency  in  basic 
skills.   Consider  a  more  extensive  range  of  criteria 
and  measures. 

PPAI/OTHER  SPECIALTIES 

22.  Review  the  several  instruments  for  principals  and 
other  specialty  personnel  for  parallel  coverage  and 
structure.   Improve  the  consistency  in  format  and 
comprehensiveness  wherever  possible. 

PRINCIPALS  AS  EVALUATORS 

23.  Assign  greater  weight  to  the  performance  of  principals 
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as  teacher  evaluators  than  to  other  functions  in  the 

PPAI  as  one  way  to  improve  the  TPAS . 
EXTRA  RESOURCES  FOR  TPAI  REVISION 
24.  Provide  extra  assistance  for  the  Department  of  Public 

Instruction  to  help  implement  the  2  3  recommendations 
above  during  summer  1988. 
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INTRODUCTION 

A  panel  of  four  teacher  evaluation  specialists  met  in 
Raleigh  from  May  8-12,  1988  to  review  the  North  Carolina 
Teacher  Performance  Appraisal  Instrument  (TPAI)  and  the 
Principal  Performance  Appraisal  Instrument  (PPAI) . 
Members  of  the  panel  were  Drs.  Richard  M.  Brandt 
(chairman) ,  Daniel  Duke,  Russell  French,  and  Edward 
Iwanicki. 

The  charge  by  the  Education  Subcommittee  of  the  Joint 
Legislative  Commission  on  Government  Operations  included: 
o    evaluating  the  TPAI  and  PPAI 
o    giving  an  opinion  on  the  fairness  and  objectivity 

of  these  instruments 
o    giving  an  opinion  whether  the  currently  adopted 
requirements  established  for  and  the  distinction 
between  career  levels  for  teachers  and  principals 
are  appropriate 
Before  and  during  these  five  days,  panel  members 
studied  several  dozen  documents  about  the  TPAI  and  PPAI 
including  legislation,  manuals  describing  the  instruments 
and  procedures  for  their  use,  training  materials,  studies 
that  show  the  distribution  of  summative  ratings  across 
career  development  pilot  districts,  surveys  of  teacher  and 
principal  reactions,  memoranda  and  directives  from  the 
Department  of  Public  Instruction,  local  district  forms  and 


reports,  and  notes  made  by  the  chairman  while  conferring 
with  superintendents,  career  development  program  (CDP) 
coordinators,  outside  evaluators  (OEs) ,  principals,  and 
teachers  during  visits  to  several  CDP  pilot  districts. 
Between  May  9  and  May  12,  the  panel  met  with  the 
subcommittee  on  two  occasions,  the  first  to  clarify  the 
charge  and  the  second  to  provide  an  overview  of  findings. 
The  panel  also  net  twice  with  DPI  staff  for  approximately 
two  hours  each  to  ask  questions  about  the  instruments  and 
the  many  documents  we  were  studying. 

Needless  to  say,  the  performance  appraisal  system  is 
complicated.   It  involves  all  of  the  certified  school 
personnel  in  North  Carolina.   Many  features  must  be 
reviewed  and  many  factors  considered  if  one  is  to  provide 
a  thorough  and  sound  evaluation  of  such  a  comprehensive 
system. 

Before  reporting  our  findings  and  recommendations,  we 
wish  to  make  three  general  observations: 

1.  We  commend  the  North  Carolina  legislature  and  the 
Education  Subcommittee  in  particular  for  recognizing  the 
need  for  a  strong  teacher  evaluation  system  and  for 
providing  the  leadership  to  see  it  established.   Strong 
leadership  and  continued  attention  will  be  necessary  to 
make  sure  it  functions  well  and  truly  serves  to  improve 
the  education  of  North  Carolina's  young  people. 

2.  We  wholeheartedly  support  the  incremental  approach 


the  Subcommittee  has  taken  to  pilot  test  the  TPAS  and  CDP 
carefully  before  going  statewide  with  them.   Implementing 
the  career  ladder  in  pilot  districts  over  a  four-year 
period  provides  a  unique  opportunity  to  fine-tune  the 
instruments  and  procedures  as  they  are  being  developed  and 
field  tested.   What  the  TPAI  has  become  as  a  result  of 
careful,  thorough  development  over  a  several  year  period 
is  a  quality  instrument,  we  all  agree.   What  we  will 
recommend  should  strengthen  it  even  further  and  help  in 
that  fine-tuning  process. 

3.    We  also  want  to  commend  the  educators  with  whom 
we  have  talked  at  both  the  state  and  district  levels.  They 
have  been  gracious  with  their  time,  candid,  and  most 
helpful  in  providing  the  numerous  materials  we  requested 
and  in  answering  our  many  questions.   We  respect  them  for 
their  great  dedication  and  effort  to  implement  these 
programs  well.   It  is  no  easy  task,  and  they  should  be 
congratulated  for  their  many  accomplishments  so  far. 


PURPOSES  OF  THE  TEACHER  PERFORMANCE  APPRAISAL  INSTRUMENT 

The  TPAI  is  being  used  currently  for  the  following 
purposes : 

1.    Certification  -  To  insure  that  new  teachers  have 
the  skills  necessary  to  continue  in  the  teaching 
profession.   By  the  end  of  their  first  two  years 
in  the  profession,  beginning  teachers  must 
demonstrate  at  least  "at  standard"  performance 
with  respect  to  the  first  five  functions  of  the 
TPAI  in  order  to  be  granted  a  renewable 
continuing  certificate. 

2.  Quality  Assurance  -  To  assure  the  public  that 
only  competent  professionals  are  allowed  to 
remain  in  the  teaching  profession.   The  TPAI  is 
used  to  evaluate  all  teachers.   Tenured  teachers 
are  required  to  demonstrate  at  least  "at 
standard"  performance  on  all  eight  functions  of 
the  TPAI. 

3.  Career  Development  -  The  North  Carolina  Career 
Development  Plan  has  been  designed  "to  attract 
and  retain  the  best  people  in  teaching.  ..." 
The  TPAI  is  the  "cornerstone"  of  this 
performance-based  plan.   Teachers  must 
demonstrate  at  least  "above  standard"  performance 
on  all  functions  of  the  TPAI  before  they  can 
advance  to  a  higher  career  status  level. 


4.    Professional  Growth  and  Development  -  To  identify 
aspects  of  teaching  performance  which  need  to  be 
strengthened  or  enhanced,  and  to  monitor  teacher 
development  in  this  regard.   For  many  teachers, 
especially  beginning  teachers  and  experienced 
teachers  encountering  difficulty,  the  TPAI 
provides  rich  information  for  use  in  designing  a 
Professional  Development  Plan  (PDP)  to  foster 
growth  with  respect  to  particular  teaching 
practices. 

Is  the  North  Carolina  Teacher  Performance  Assessment 
Instrxunent  adequate  for  these  purposes? 

The  TPAI  is  adequate  for  the  purposes  of  certification 

and  quality  assurance.   The  first  five  functions  of  the 

TPAI  are  keyed  to  the  research  on  effective  instruction. 

Since  the  instructional  practices  subsumed  under  these 

functions  are  expected  of  the  competent  teacher,  they 

provide  an  adequate  basis  for  making  certification 

decisions.   Although  the  last  three  functions  of  the  TPAI 

are  less  directly  related  to  instruction,  they  do  include 

additional  teaching  practices  essential  to  teaching 

competency  within  a  particular  school  context.   Thus,  the 

eight  major  functions  of  the  TPAI  provide  a  good  basis  for 

assessing  teacher  performance  for  the  purpose  of  quality 

assurance,  especially  if  the  school  system  has  delineated 


clearly  its  expectations  for  teachers  with  respect  to 

functions  6-8. 

With  respect  to  the  final  two  purposes,  career 
development  and  professional  growth  and  development,  the 
TPAS  seems  to  be  functioning  reasonably  well  considering 
the  CDP  is  only  three  years  old.   It  needs  to  be 
strengthened  and  expanded  in  some  areas,  however,  if  these 
purposes  are  to  be  fully  achieved.   The  remaining  sections 
of  this  report  will  focus  on  how  the  TPAS  can  be 
strengthened  to  meet  these  purposes  better. 


TPAI  CRITERIA:   TEACHING  FUNCTIONS  AND  PRACTICES 

Are  the  major  functions  and  inherent  teaching  practices 
which  comprise  the  TPAI  adequate? 

Although  the  major  functions  and  teaching  practices 
comprising  the  TPAI  are  adequate  in  many  respects,  they 
could  be  improved.   Variations  in  rating  consistency, 
especially  on  functions  6,  7,  and  8,  suggest  the  need  for 
greater  clarity.   Also,  a  need  to  expand  the  functions  and 
overall  coverage  of  the  instrument  has  been  expressed  by 
many  teachers  and  administrators. 

Our  first  recommendation  is  that  some  reordering  take 
place  with  respect  to  functions  6-8.   Specifically,  it  is 
suggested  that  function  7  (Communicating  within  the 
Educational  Environment)  becomes  function  6,  and  function 
6  (Facilitating  Instruction)  becomes  function  7. 
Furthermore,  the  new  function  6  should  be  renamed 
"Communication  and  Interaction  in  the  Classroom".   To 
accomplish  this  change,  the  current  practice, "Teacher 
interacts  effectively  with  students,  coworkers,  parents, 
and  community",  would  be  divided  into  the  separate 
practices  and  placed  in  separate  functions  as  noted  below: 

6.    Teacher  interacts  effectively  with  students. 

7.6   Teacher  communicates  effectively  with  coworkers, 
parents,  and  community. 
Finally,  two  practices  from  function  3  (3.3  &  3.4)  should 
also  be  added  to  the  new  function  6,  resulting  in  the 
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following  new  function: 

6.    Communicates  and  Interacts  in  the  Classroom 

6.1  Teacher  speaks  fluently  and  precisely. 

6.2  Teacher  presents  the  lesson  or  instructional 
activity  using  concepts  and  language 
understandable  to  the  students. 

6.3  Teacher  interacts  effectively  with  students. 

6.4  Teacher  treats  all  students  in  a  fair  and 
equitable  manner. 

As  a  result  of  this  reordering,  the  first  six  functions  of 
the  TPAI  would  be  related  directly  to  instruction  and 
observable  in  the  classroom.   All  six  would  be  used  to 
make  certification  decisions.   The  final  two  functions 
would  be  related  less  directly  to  instruction  and  would 
not  be  directly  observable  in  the  classroom. 

Second  the  initial  six  functions  of  the  TPAI  dealing 
with  instruction  need  to  be  reviewed  to  determine  whether 
they  clearly  communicate  to  the  teacher  those  critical 
aspects  of  effective  teaching  identified  in  the 
literature.   During  this  process,  it  is  recommended  that 
function  3  (Instructional  Presentation)  be  subdivided  into 
a  series  of  more  specific  functions  as  indicated  below. 
Each  of  these  new  functions  would  include  a  set  of 
specific  teaching  practices  drawn  in  large  part  from 
current  function  3.   Additional  teaching  practices  for 
this  function  could  be  drawn  from  the  Connecticut 
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Competency  Instrument. 

o    Initiation  of  Instruction 
o    Motivation  of  Students 

o    Management  of  Routines  and  Transitions 
o    Presentation  of  Accurate  and  Appropriate  Content 
o    Provision  of  Closure 

If  this  recommendation  is  adopted,  the  resultant 
version  of  the  TPAI  would  consist  of  ten  rather  than  six 
functions  dealing  with  instruction  which  could  be  assessed 
through  classroom  observation.   These  functions  would 
include  the  following: 

o    Management  of  Instructional  Time 
o    Management  of  Student  Behavior 
o    Initiation  of  Instruction 
o    Motivation  of  Students 

o    Presentation  of  Accurate  and  Appropriate  Content 
o    Management  of  Routines  and  Transitions 
o    Instructional  Monitoring  of  Student  Performance 
o    Instructional  Feedback 

o    Communication  and  Interaction  in  the  Classroom 
o    Provision  of  Closure 
This  expanded  number  of  functions  dealing  with  instruction 
would  strengthen  the  validity  and  reliability  of  the  TPAI. 
Validity  would  be  enhanced  since  ten  functions  sample  more 
comprehensively  the  dimensions  of  effective  teaching 
presented  in  the  literature  (Good  and  Brophy,  1987; 


Wittrock,  1986) .   Reliability  would  be  enhanced  since  the 

TPAI  could  now  measure  more  aspects  of  effective  teaching 

performance  in  a  highly  focused  manner. 

It  is  foreseen  that  the  recommendation  to  include  a 

function  for  Presenting  Accurate  and  Appropriate  Content 

in  the  TPAI  will  be  questioned,  since  an  argument  has  been 

developed  for  not  evaluating  content  knowledge  through  the 

appraisal  process  (see  letter  of  2/15/88  to  Principals  and 

Teachers  from  R.D.  Boyd).   We  do  not  intend  to  imply  a 

deficit  model,  as  suggested  in  the  above  letter,  but 

rather,  to  include  a  dimension  of  instruction  which  is 

essential  to  the  teacher  appraisal  process.   To  omit  a 

reference  to  this  function  could  prevent  a  principal  from 

dealing  with  inadequate  treatment  of  content  by  a  teacher. 

In  appraising  teaching  performance  with  respect  to  this 

function,  the  evaluator  would  focus  on  at  least  the 

following  practices: 

o    Teacher  presents  content  without  error  or 
misinformation . 

o    Teacher  presents  content  which  is  developmental ly 
appropriate  for  the  class 

In  addition  to  expanding  the  number  of  functions 

related  directly  to  instruction,  the  reliability  and 

validity  of  the  TPAI  could  be  strengthened  through  a 

review  of  the  teaching  practices  included  in  each 

function.   With  respect  to  validity,  it  is  important  that 

these  practices  reflect  the  current  literature  on 
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effective  teaching.   While  the  practices  included  in  the 
TPAI  are  consistent  with  the  process-product  research,  the 
direct  instruction  model,  and  the  six-step  lesson  plan,  it 
would  be  helpful  to  consider  other  well-researched  and 
effective  models  of  teaching,  such  as  the  coaching, 
modeling,  cooperative,  and  mastery  learning  models,  when 
identifying  teaching  practices  to  be  included  within  each 
function.   While  parsimony  is  always  an  important 
consideration  when  selecting  teaching  practices  for  a 
particular  function,  it  should  not  preclude  adding 
practices  which  enhance  the  validity  of  the  TPAI.   We  also 
recommend  adding  to  Function  2  the  following  practice — 
"teacher  reinforces  appropriate  behavior."  Allowing  this 
practice  to  become  2.6  would  increase  the  validity  of  the 
Management  of  Student  Behavior  function. 

The  reliability  of  the  TPAI  is  enhanced  by  the  extent 
to  which  the  teaching  practices  for  each  function  are 
stated  clearly  and  thus  are  interpreted  in  the  same 
manner.   For  example,  the  Management  of  Student  Behavior 
Function  includes  the  practice — "Teacher  stops 
inappropriate  behavior  promptly  and  consistently,  yet 
maintains  the  dignity  of  the  student."  This  practice 
could  be  interpreted  differently  by  various  observers. 
Some  might  expect  the  teacher  to  actually  stop  the 
behavior,  while  others  might  expect  the  teacher  to  deal 
with  the  problems  appropriately,  which  could  include 
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ignoring  the  behavior.   The  differing  interpretations  of 
this  practice  would  contribute  to  the  unreliability  of 
rating  teachers  on  this  function. 

The  intent  here  is  not  to  imply  that  there  are 
substantial  gaps  between  the  literature  on  effective 
teaching  and  the  practices  included  in  the  TPAI,  nor  to 
create  the  impression  that  there  is  considerable  ambiguity 
in  the  manner  in  which  these  practices  are  described.   For 
the  most  part  there  is  good  match  between  the  literature 
on  effective  teaching  and  the  practices  in  the  TPAI. 
Also,  these  practices  tend  to  be  described  clearly.   The 
point  being  made  here  is  that  the  TPAI  is  not  a  static 
instrument.   It  must  be  reviewed  periodically  to  insure 
that  the  practices  included  are  consistent  with  the 
current  knowledge  base  regarding  effective  teaching,  and 
that  these  practices  are  being  interpreted  in  a  similar 
manner  by  the  parties  using  the  TPAI  and  affected  by  its 
outcomes.   The  need  for  such  review  provides  a  prime 
opportunity  to  involve  teachers  and  principals  in  any 
revisions  of  the  appraisal  instrument.   A  committee 
structure  could  be  organized  whereby  teachers  and 
administrators  are  actively  involved  in  the  process  of 
updating  the  TPAI.   Through  this  approach,  the  quality  of 
the  TPAI  would  be  sharpened  while  building  ownership  among 
school  practitioners  in  the  revised  appraisal  process. 

To  this  point,  the  focus  has  been  on  those  aspects  of 
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the  TPAI  directly  related  to  instruction.   As  a  result  of 
the  revisions  suggested,  two  TPAI  functions  remain  which 
are  related  less  directly  to  instruction.   These  are 
o    Facilitating  Instruction 
o    Performing  Non-Instructional  Duties 
It  is  clear  that  there  is  considerable  ambiguity  as  to  how 
teachers  should  be  evaluated  with  respect  to  these 
functions.   While  it  is  important  for  the  DPI  to  provide 
some  guidance  in  this  regard,  it  also  is  critical  that 
school  districts  take  leadership  in  identifying  those 
practices  their  teachers  are  expected  to  pursue  with 
respect  to  these  functions.   To  a  large  extent  LEAs  are  at 
liberty  to  customize  these  two  functions  to  their  local 
needs.   In  doing  so,  practices  may  be  included  which  a.) 
add  a  dimension  of  ingenuity  and  creativity  to  the 
appraisal  process,  and  b.)  begin  to  address  the  affective 
component  of  the  educational  process.   To  date  these  areas 
have  not  been  addressed  sufficiently  through  the  appraisal 
process. 

Those  who  undertake  the  further  development  of  these 
functions  at  either  the  local  or  state  level  should 
realize  that  only  evaluation  criteria  derived  from  one  of 
two  sources  will  stand  up  to  appeal  or  legal  challenge: 
(a)   research  findings  or  (b)  consensus  agreement  of  those 
subject  to  them.   Since  there  is  little  research  to 
support  many  practices  which  could  be  included  under  these 
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two  functions,  evidence  should  be  obtained  on  either  a 
local  or  statewide  basis  that  most  teachers  agree  with  the 
inclusion  of  these  practices  (French  and  Malo,  1987). 

This  section  has  focused  on  a  discussion  of 
considerations  in  reordering  and  expanding  the  functions 
currently  included  in  the  TPAI .   The  direction  advocated 
would  strengthen  the  reliability  and  validity  of  the  TPAI 
for  naking  certification  and  career  status  I  and  II 
decisions.   Later  in  this  report  a  series  of  alternatives 
for  making  career  status  III  decisions  will  be  addressed. 
One  of  these  alternatives  is  simply  to  use  the  TPAI.   If 
the  TPAI  is  to  be  used  in  this  manner,  additional 
functions  would  need  to  be  added.   Such  functions  must  be 
consistent  with  an  expanded  view  of  teaching  applicable  at 
the  career  status  III  level.   If  this  view  encompasses  the 
role  of  teacher  as  instructional  leader,  then  the 
instructional  leadership  functions  of  the  PPAI  might  be 
adapted  to  the  TPAI  for  use  in  making  career  status  III 
decisions.   More  specifically,  the  functions  to  be  added 
to  the  TPAI  might  include  the  following: 

Develops  a  comprehensive  instructional  plan 
Implements  the  comprehensive  instructional  plan 

fnSt^i:??^  ^?^  delivery  of  the  comprehensive 
instructional  plan 

°    fn^^""^^?^  ^^f  ^"'P^''^  °^  ^^^  comprehensive 
instructional  plan 

The  practices  included  in  the  PPAI  for  each  of  these 
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functions  would  also  need  to  be  reviewed  and  adopted  for 
use  in  the  expanded  version  of  the  TPAI . 

In  summary,  the  functions  and  inherent  practices  that 
currently  comprise  the  TPAI  tend  to  be  supported  by  the 
literature  on  effective  teaching.   This  is  true 
particularly  for  the  first  five  functions  which  are 
related  directly  to  instruction.   To  enhance  the 
reliability  and  validity  of  the  TPAI  in  making 
certification  and  career  status  I  and  II  decisions  it  is 
recommended  that  a.)  the  functions  of  this  instrument  be 
reordered,  b.)  some  functions  be  expanded,  and  c.)  the 
practices  within  each  function  be  reviewed  in  light  of  the 
most  recent  literature  on  models  of  teaching  and  effective 
instruction.   If  these  recommendations  are  followed,  the 
revised  version  of  the  TPAI  would  assess  the  functions 
noted  below. 

Functions  which  can  be  assessed  through  classroom 
observation: 

*-   Management  of  Instructional  Time 

Initiation  of  Instruction 

Presentation  of  Accurate  and  Appropriate  Content 

Motivation  of  Students 

Communication  and  Interaction  in  the  Classroom 

Management  of  Routines  and  Transitions 
*-   Management  of  Student  Behavior 
*-   Instructional  Monitoring  of  Student  Performance 
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*-    Instructional  Feedback 
Provision  of  Closure 

Functions  which  would  be  assessed  primarily  in  other 
ways  and  with  considerable  local  specification  of 
criteria: 

*  Facilitating  Instruction 

*  Performing  Non-Instructional  Duties 


♦Functions  included  in  the  current  TPAI 


16 


PROCEDURES  AND  INSTRUMENTS 

Are  the  procedures  and  instruments  used  for 
collecting,  analyzing  and  interpreting  data  sufficient  a.) 
for  the  development  of  fair  and  equitable  personnel 
decisions  and  b.)  for  the  improvement  of  instruction? 

To  respond  to  the  opening  question,  four  sub- 
questions  must  be  addressed: 

1.  Are  instruments  used  appropriate  to  the  purposes 
of  the  system  and  the  established  criteria?  Are 
they  fair  and  equitable  to  all  persons  to  whom 
they  are  applied? 

2.  Are  those  administering  the  system  (those 
observing  and  evaluating)  adequately  trained? 

3.  Is  the  use  of  the  system  for  summative  purposes 
well  coordinated  and  consistently  conducted  from 
one  time  and  place  to  another? 

4.  Are  the  instruments  being  used  effectively  in 
formative  ways  to  assist  teachers  and  improve 
instruction? 

Each  of  these  four  sub-questions  will  be  treated 

separately. 

Are  the  instruments  used  appropriate  to  the  purposes 
of  the  system  and  the  established  criteria?  Are  they  fair 
and  equitable  to  all  persons  to  whom  they  are  applied? 

The  purposes  to  which  TPAS  results  are  applied  include 

continuing  certification,  adequacy  of  continuing 

performance  and  determination  of  level  of  teaching 

performance  as  defined  for  career  ladder  placement.   The 

eight  functions  and  38  related  practices  presently  defined 
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within  the  TPAS  (or  any  revised  version  thereof) 
constitute  the  criteria  upon  which  instrumentation  and 
data  collection  procedures  must  focus.   At  present,  the 
actual  instruments  contained  within  the  TPAS  include  the 
Teacher  Performance  Appraisal  Instrument  (TPAI) ,  rating. 
scale  descriptors  necessary  to  its  use,  and  the 
Professional  Development  Plan  Form  (PDP-1) .   These 
instruments  are  supported  by  two  others  created  to 
systematize  the  collection  of  classroom  observation  data 
(the  Formative  Observation  Data  Instrument)  (FODI)  and 
their  application  to  the  evaluation  criteria  (Formative 
Observation  Data  Analysis)  (FODA) . 

Furthermore,  the  1985  legislation  establishing  the 
School  Career  Development  Pilot  Program  specifies  that 
multiple  sources  of  information  will  be  used  to  evaluate 
candidates  for  career  ladder  placement.   It  further 
specifies  that  candidates  for  career  status  II  will  submit 
for  analysis  a  portfolio  of  work-related  documents  for 
analysis  and  that  an  interview  with  the  candidate  can  be 
used  as  yet  another  source  of  data. 

It  is  clear  that  the  instruments  used  as  a  basis  for 
assessing  performance  for  certification  purposes  (FODI, 
FODA,  TPAI)  are  appropriate  to  that  purpose.   The  first 
five  teaching  functions  to  be  evaluated  within  the  TPAS 
are  strongly  supported  by  effective  teaching  research  and 
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are  observable  in  the  classroom  setting.*   The  practices 
identified  within  those  functions  represent  basic 
pedagogical  skills  which  teachers  should  be  able  to 
demonstrate  in  their  classrooms  as  a  condition  of 
licensure.   At  this  time,  it  is  these  first  five  functions 
and  practices  which  are  evaluated  and  used  as  a  basis  for 
recommending  the  issuance  of  a  continuing  certificate  in 
North  Carolina.   The  combination  of  the  FODI,  FODA  and 
TPAI  provides  an  appropriate  means  of  gathering  and 
analyzing  teacher  performance  data  pertinent  to  this 
decision  for  most,  if  not  all,  teachers.   However,  the 
validity  and  reliability  of  the  decision  process  can  be 
improved  by  modifying  TPAS  criteria  as  recommended. 

Obviously,  the  annual  performance  evaluations  required 
of  all  teachers  in  North  Carolina  are  perceived  to  be  the 
primary  means  of  a.)  assuring  that  the  quality  of  a 
teacher's  performance  remains  constant  at  or  above 
standard  as  defined  within  the  structure  of  the  state- 
approved  evaluation  system  (TPAS)  and  b.)  providing 
professional  development  assistance  to  both  those  who  need 
improvement  and  those  who  desire  to  continually  improve 
their  teaching  performance.   Again,  the  instruments 
primarily  used  for  these  purposes  are  the  FODI,  FODA  and 


*This  statement  will  be  even  more  true  if 
modifications  to  the  criteria  suggested  earlier  in  this 
report  are  made. 
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TPAI  which  are  appropriate  to  assessment  of  performance 
of  observable,  instructional  functions  (such  as  current 
TPAS  functions  one  through  five) .   However,  these 
instruments  alone  are  not  sufficient  to  measure  non- 
instructional,  non-observable  skills  such  as  those 
embodied  in  the  present  TPAS  functions  6,  7  and  8, 
although  they  do  contribute  some  pertinent  information. 
At  the  present  time,  there  is  little  evidence  of  either 
state-developed  or  locally-developed  instruments  which 
have  been  adopted  statewide  to  supplement  the  FODI ,  FODA 
and  TPAI.   There  is  evidence  to  suggest  that  some  local 
school  districts  have  developed  and  are  using  such 
instruments,  but  there  has  been  no  systematic  statewide 
attempt  to  address  this  problem. 

The  School  Career  Development  Pilot  Program  has 
created  a  new  set  of  teacher  evaluation  issues  to  be 
addressed,  and  logically,  the  TPAS  is  being  modified  and 
adapted  to  address  some  of  those  issues.   Since  the 
legislation  and  funding  undergirding  the  Career 
Development  Pilot  Program  is  only  three  years  old,  both 
the  State  Department  of  Public  Instruction  and  the  sixteen 
local  districts  involved  in  pilot  programs  have  had  to 
confront  and  implement  numerous  policy  and  technical 
decisions  in  a  very  short  amount  of  time.   There  has  not 
yet  been  enough  time  to  create  all  that  needs  to  be 
created,  to  identify  all  the  "bugs"  in  evaluation 
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instruments  and  procedures  or  to  fine-tune  and  systematize 
the  evaluation  system.   The  efforts  of  all  those  who  have 
been  involved  in  program  development  and  implementation 
are  to  be  highly  commended  and  applauded. 

As  stated  earlier  in  this  document,  some  of  the 
purposes  defined  for  the  evaluation  system  to  be  used  in 
Career  Development  Program  decisions  are  to 

o    encourage  differentiation  of  all  teachers  and 
administrators  (differentiate  among  levels  of 
performance — as  currently  defined  career  status  I 
and  II) 

o    encourage  recognition  of  high  quality  teachers 
(award  career  status  II) 

o    improve  the  quality  of  classroom  instruction 
It  does  not  appear  that  the  instruments  currently 
incorporated  in  the  TPAS  are  fully  adequate  for  achieving 
all  of  these  purposes.   However,  this  statement  should  not 
be  interpreted  to  mean  that  the  present  TPAS  structure  and 
instruments  should  be  discarded.   Modification  and 
supplementation,  not  discard,  are  needed,  and  those 
persons  in  the  Department  of  Public  Instruction  who  are 
charged  with  development  and  implementation  of  the  program 
are  already  aware  of  and  have  clearly  stated  a  number  of 
those  needs. 

As  noted  previously,  the  current  instruments  (FODI, 
FODA,  TPAI)  are  not  alone  sufficient  to  assess  performance 
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in  all  functions  of  the  TPAS.   Functions  6,  7  and  8,  in 
either  their  present  or  modified  form,  require  the 
collection  and  rating  of  information  from  evaluators 
either  in  the  form  of  work  products  or  oral  descriptions 
of  activities  (data  sources  acknowledged  in  the  1985 
legislation) . 

Examination  of  the  current  instruments  in  TPAS  (FODI, 
FODA,  TPAI  and  the  related  rating  scale  PDP-1)  reveals 
that  there  are  refinements  which  could  be  made  in  each 
instrument  to  enhance  its  capacity  to  a.)  differentiate 
between  career  status  I  and  II  performance  and  b.)  improve 
the  quality  of  classroom  instruction  (the  teacher's  growth 
toward  mastery  of  the  desired  teaching  functions  and 
practices) .   In  the  context  of  these  analyses  of  the 
evaluation  task  and  present  and  "missing"  instruments, 
several  recommendations  are  offered. 
Formative  Observation  Data  Instrument  (FODI) 

The  Formative  Observation  Data  Instrument  provides 
descriptive  information  about  the  teacher's  classroom 
performance.   Since  it  is  an  instrument  which  requires  the 
observer  to  simply  record  what  he/she  actually  sees  and 
hears  without  rendering  judgment,  it  appears  to  have  the 
capacity  to  treat  all  who  are  observed  objectively,  fairly 
and  equitably.   However,  other  forms  of  descriptive  data 
could  be  used  to  supplement  these  scripts  in  the  effort  to 
provide  an  evaluatee  specific  information  which  will 
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enable  him/her  to  improve  performance  and  at  the  same  time 
sharpen  the  ability  of  evaluators  to  discern  and  document 
differences  in  performance.   For  example,  mapping  of 
student  and  teacher  movement  in  the  classroom  or  making 
charts/scans  of  student  on-task  and  off-task  behavior  are 
powerful  tools  for  pinpointing  specific  teacher  strengths 
and  weaknesses  related  to  certain  of  the  TPAS  criteria. 
Obviously,  an  observer  cannot  record  several  kinds  of 
information  simultaneously,  particularly  when  scripting  is 
used.   But  during  a  50-minute  observation,  a  pattern  of 
different  types  of  data  gathering  can  be  easily  instituted 
without  losing  major  teacher  or  student  behaviors.   For 
instance,  one  night  employ  a  sequence  of  descriptive  data- 
gathering  activities  such  as  the  following: 

Script  classroom  events  (FODI)  -  10  minutes 
Map  teacher/student  movement  -  3  minutes 
Scan  student  off-task  behavior  -  2  minutes 
Script  classroom  events  (FODI)  -  10  minutes 
Map  teacher/student  movement  -  3  minutes 
Scan  student  off-task  behavior  -  2  minutes 
We  would  recommend  that  consideration  be  given  to 
supplementing  the  script  data  collected  with  the  FODI,  but 
we  are  not  prepared  to  say  exactly  what  additional 
instrumentation  or  procedures  should  be  used  or  even  that 
they  absolutely  should  be  used.   There  are  a  number  of 
technical,  procedural  and  policy  issues  which  will  need  to 
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be  addressed  in  such  considerations. 

In  keeping  with  some  of  the  recommended  changes  in 
TPAS  criteria  presented  earlier  in  this  document,  we  would 
recommend  that  teachers  be  provided  copies  of  all 
descriptive  data  (FODI's  and  any  other  descriptive 
instruments  used)  in  advance  of  the  post-observation 
conference  with  an  observer.   This  procedure  would  allow 
the  evaluatee  to  do  some  self-evaluation  and  should  make 
the  post-observation  conference  more  productive, 
especially  in  regard  to  professional  development  planning. 
Formative  Observation  Data  Analysis  (FODA) 

Although  the  Formative  Observation  Data  Analysis 
instrument  uses  the  word  formative  in  its  title,  it 
embodies  both  formative  and  summative  dimensions.   One 
purpose  of  the  instrument  clearly  is  to  summarize  and 
synthesize  the  raw  data  contained  in  the  FODI .   However, 
users  also  are  instructed  to  "use  statements  which 
accurately  reflect  the  quality  of  performance  documented 
by  your  raw  data."   In  purely  formative  evaluation,  the 
evaluator  and  evaluatee  generally  complete  some  kind  of 
data  analysis  and  professional  development  plan  mutually 
during  the  post-observation  conference.   The  FODA  seems 
designed  to  fulfill  this  function  and  to  serve  as  the 
middle  step  (hinge)  between  observation  data  and  summative 
evaluation  (performance  judgments  by  evaluators) .   Since 
both  functions  must  be  served  in  fulfilling  the  multiple 
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uses  of  the  TPAI  and  its  related  instruments,  there  is 
probably  no  good  alternative  to  the  tensions  inherent  in 
this  instrument.   However,  both  evaluators  and  evaluatees 
should  be  aware  of  those  tensions  and  every  care  should  be 
taken  to  address  both  the  formative  and  summative 
dimensions  of  the  instrument  as  professionally  as 
possible. 

The  FODA  contains  no  space  for  an  evaluatee  to  respond 
to  the  observations  and  quality  assessments  made  by  an 
evaluator.   While  the  evaluatee  is  invited  to  respond  on  a 
separate  sheet  of  paper,  providing  space  on  the  form  would 
signal  greater  interest  in  teacher  input. 

The  dual  purposes  of  the  FODA  require  that  every 
evaluator  understand  the  instrument  and  be  well  trained  in 
completing  and  using  it.   The  Department  of  Public 
Instruction  has  made  available  both  initial  and  "booster" 
training  in  FODA  completion  and  several  local  school 
districts  have  further  emphasized  this  component  of  the 
evaluation  process.   However,  there  does  not  appear  to  be 
sufficient  training  for  evaluators  (principals  and 
observer  evaluators)  in  conferencing  and  in  using  the  FODI 
and  FODA  in  the  conference  setting. 
Teacher  Performance  Appraisal  Instrument  and  Rating  Scale 

The  Teacher  Performance  Appraisal  Instrument  is  a 
summative  instrument  in  which  data  generated  from  other 
instruments  and  sources  are  aggregated  and  translated  into 
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a  set  of  performance  ratings.   The  primary  vehicle  for 
this  translation  is  the  six-point  rating  scale. 

The  panel  is  aware  that  there  have  been  requests  that 
the  current  six-point  rating  scale  be  reduced  to  three 
points.   This  reduction  should  not  be  made  because  it 
would  lessen  greatly  the  capacity  of  the  TPAI  to 
differentiate  among  levels  of  performance  -  an  important 
intended  outcome  of  the  Pilot  Career  Development  Program. 
The  use  of  a  three-point  scale  would  reduce  the  instrument 
to  one  which  is  capable  of  depicting  only  minimal 
competence  (satisfactory  or  unsatisfactory  performance) . 

The  intervals  between  points  on  the  rating  scale  are 
unequal,  that  is,  the  distance  between  critical  points  2 
and  3  is  greater  than  the  distance  between  points  3-4,  4- 
5,  5-6.   This  feature  was  consciously  designed  into  the 
scale.   However,  it  may  be  contributing  somewhat  to  lack 
of  reliability  among  raters.   Experimentation  with  a 
multi-stage  decision-making  process  has  been  conducted, 
and  the  panel  recommends  that  this  process  be  implemented 
statewide.   In  essence,  this  recommendation  means  that  an 
evaluator  will  first  decide  whether  or  not  the  teacher's 
overall  performance  on  a  function  is  at  standard 
(adequate)  or  below  standard  (inadequate).   If  performance 
is  judged  to  be  at  least  at  standard,  the  next  decision 
will  be  whether  the  performance  is  at  standard,  or 
somewhere  above  standard.   If  the  performance  is  somewhere 
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above  standard,  the  evaluator  must  next  decide  whether  the 
rating  should  be  a  4  (above  standard)  or  a  5  or  6.   In 
reality,  the  standards  currently  operable  in  determining 
career  status  through  level  II  (2  ratings  at  least  5,  5 
ratings  at  least  4,  1  rating  at  least  3)  dictate  that 
evaluators  must  be  especially  sensitive  to  the 
distinctions  between  2  and  3,  3  and  4,  and  4  and  5.   If 
the  standards  are  changed,  or  if  career  status  III  is 
added  to  the  Career  Development  Program  with  higher 
performance  standards  required,  our  recommended  procedure 
for  use  of  the  scale  will  also  need  to  be  adjusted. 

Some  attention  should  be  given  to  further 
clarification  and  elaboration  of  rating  scale  definitions 
and  particularly  to  refining  and  extending  the  training  of 
evaluators  in  recognizing  distinctions  in  performance 
data.   Both  activities  will  help  to  improve  rater 
reliability. 
Professional  Development  Plan  fPDP) 

The  Professional  Development  Plan  form  is  the  major 
instrument  for  targeting  and  directing  professional  growth 
activities.   The  form  itself  is  well  designed.   However, 
it  should  be  recognized  that  the  plan  targets  growth  only 
within  the  context  of  the  TPAS  standards;  i.e.,  growth 
toward  competence.   While  this  type  of  growth  is  needed 
and  especially  important  in  the  formative  evaluation  of 
individuals  seeking  continuing  certification  and  in 


27 


continuing  evaluations  to  assure  quality  of  basic 
performance,  teachers  who  are  already  well  above  standard 
or  superior  in  their  performance  of  basic  teaching  skills 
need  to  plan  professional  development  beyond  competence. 
The  Professional  Development  Plan  form  could  be  slightly 
modified  for  use  in  Career  Development  evaluations  to 
reflect  this  perspective. 
Need  for  Additional  Instruments 

As  indicated  in  previous  statements,  at  least  one 
additional  instrument  is  needed — a  form  and  procedure  for 
recording  and  analyzing  portfolio  contents  which  can  be 
used  statewide  to  bring  some  consistency  to  the  analysis 
of  these  required  materials.   Once  this  instrument  is 
developed  or  adopted  from  among  instruments  already 
existent  in  local  school  districts,  evaluators  will  need 
to  be  trained  in  its  use. 

A  second  instrument  should  be  considered — an  interview 
format.   Such  instruments  will  need  to  be  developed  for 
use  in  the  evaluations  of  principals  and  special  groups 
(e.g.,  counselors,  school  psychologists,  etc.),  and  a 
teacher  interview  would  be  most  useful  in  collecting  and 
interpreting  data  pertinent  to  assessment  of  those 
functions  and  practices  which  go  beyond  the  classroom.   If 
a  teacher  interview  is  constructed,  it  is  recommended  that 
it  contain  both  a  structured  section  (standard  questions 
to  be  asked  of  all  teachers)  and  an  unstructured  section 
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(a  section  in  which  a  teacher  can  provide  information 
pertinent  to  evaluation  criteria  which  he/she  feels  to  be 
important) .   A  time  for  unstructured  input  will  help  to 
involve  teachers  more  fully  in  the  evaluation  process  and 
overcome  their  concerns  that  they  have  little  opportunity 
to  provide  data  which  they  consider  to  be  important. 
Obviously,  evaluators  will  need  to  be  trained  to  conduct 
both  parts  of  the  interview,  record  pertinent  data  and 
interpret  them. 

Some  examples  of  items  or  instruments  which  could  be 
adopted  to  use  for  either  the  portfolio  analysis  or  an 
interview  protocol  have  been  developed  in  the  Pilot  Career 
Development  Program  districts.   For  example,  Harnett 
County  has  a  portfolio  analysis  form  and  a  questionnaire 
focused  on  Functions  6,  7  and  8  which  could  be  used  or 
modified  for  use  in  an  interview  setting.   Teaching 
practices  which  lend  themselves  to  data  gathering  through 
the  interview  process  are  6.1,  6.2,  6.5,  and  7.2. 
Portfolio  contents  should  provide  information  pertinent  to 
current  practices  6.1,  6.3,  6.4  and  8.1-8.3  if  appropriate 
documents  are  required  for  portfolio  inclusion. 

Near  the  beginning  of  this  section,  we  stated  that  the 
instruments  now  in  use  appear  to  be  appropriate  to  most, 
if  not  all,  teachers  to  whom  they  are  being  applied.   In 
Tennessee's  Career  Ladder  Program  and  in  evaluation 
programs  in  several  school  districts  (e.g.,  Warren,  Ohio; 
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Youngstown,  Ohio;  Dundee,  Illinois) ,  it  has  been  found 
that  some  instruments  (and  even  a  few  criteria)  used  to 
measure  the  performance  of  teachers  in  typical  classrooin 
settings  do  not  lend  themselves  to  certain  special 
education  settings  where  teachers  work  primarily  one-on- 
one  with  severely  handicapped  students,  and  they  do  not 
yield  sufficient,  appropriate  data  upon  which  to  base 
evaluative  decisions.   It  should  be  noted  that  the  same 
problem  does  not  exist  in  resource  classes  or  other 
settings  where  the  student  population  is  composed  of 
mildly  handicapped  learners.   These  findings  in  other 
settings  lead  us  to  recommend  that  careful  monitoring  of 
the  evaluation  criteria,  procedures  and  instruments  in 
these  particular  unique  settings  be  conducted,  and  that 
appropriate  changes  be  made  if  evaluation  data  and 
evaluator  perceptions  suggest  that  they  are  needed. 
Training  of  Evaluators 

Are  those  administering  the  system  (those  observing 
and  evaluating)  adequately  trained? 

In  applying  the  Teacher  Performance  Appraisal  System 

statewide  for  the  several  purposes  previously  discussed, 

there  are  essentially  four  groups  of  people  who  are 

evaluating  teachers:   principals,  mentor  teachers, 

observer/evaluators  and  "others"  who  may  serve  on  support 

teams  in  some  school  districts.   At  present,  there  are 

four  types  of  training  which  impacted  upon  either  the 

formative  or  summative  dimensions  of  the  Teacher 
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Performance  Appraisal  System:   Effective  Teaching 
Training,  Teacher  Performance  Appraisal  Training, 
Professional  Development  Plans  and  Mentor/Support  Team 
Training.   All  of  these  training  programs  operate  on  the 
turnkey  principle;  i.e.,  the  Department  of  Public 
Instruction  trains  trainers  for  local  school  districts  who 
in  turn  train  appropriate  others. 

The  Department  of  Public  Instruction  with  the 
assistance  of  numerous  educators  from  local  school 
districts  and  higher  education  institutions  has  done  an 
excellent  job  in  developing  these  training  programs  and 
related  materials  and  making  the  training  available, 
especially  when  one  considers  they  have  had  to  "fly  the 
plane  while  still  building  it."   However,  the  review  panel 
has  found  some  problems  which  need  to  be  addressed  if 
evaluation  processes  and  decisions  are  to  be  highly  valid 
and  reliable: 

1.    Although  the  vast  majority  of  teachers  and 
principals  in  the  state  have  experienced 
Effective  Teaching  Training  or  its  equivalent,  it 
is  possible  that  there  are  evaluators  within  the 
four  identified  groups  who  have  not  been  through 
this  experience  or  mastered  this  knowledge  which 
is  essential  to  understanding  the  TPAS  functions 
and  practices  and  to  coaching  teachers  for 
improvement  in  these  performance  areas. 
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2.    Not  all  r.entors  and  others  who  evaluate  (and 

perhaps  even  principals  and  assistant  principals) 
have  been  required  to  take  the  Teacher 
Performance  Appraisal  Training  and/or  demonstrate 
competence  in  teacher  appraisal. 

3.  It  is  possible  that  there  are  persons  evaluating 
who  have  not  had  the  training  in  Professional 
Development  Planning, 

4.  All  who  evaluate  must  be  prepared  to  give 
feedback,  coach  and  support  evaluators,  but  may 
not  have  had  the  Mentor/Support  Team  Training. 

5.  A  review  of  training  materials  in  the  various 
training  programs  indicates  that  only  the 
Mentor/Support  Team  Training  emphasizes  the 
development  of  conferencing  skills,  yet  every 
evaluator  must  be  effective  in  conferencing  if 
formative  evaluation  (evaluation  for  improvement) 
is  truly  to  take  place. 

It  cannot  be  assumed  that  any  evaluator— principal, 
central  office  administrator,  higher  education  faculty 
member  or  experienced  teacher-has  the  skills  necessary  to 
carry  out  the  many  evaluation-related  activities. 
However,  it  is  possible  that  individuals  have  gained  those 
skills  in  settings  other  than  the  training  programs 
developed  and  implemented  by  the  Department  of  Public 
instruction.   Therefore,  the  review  panel  recommends  that 
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a  process  be  developed  for  certifying  these  competencies 
of  all  persons  who  evaluate  teachers  in  a.)  the  research 
on  effective  teaching,  b.)  teacher  appraisal  using  the 
TPAS  structure  and  instruments,  professional  development 
planning,  and  effective  conferencing.   Mentors  should  also 
be  required  to  demonstrate  competence  in  the  additional 
knowledge  and  skills  required  of  a  mentor.   Persons  who  do 
not  meet  the  certification  requirements  should  not  be 
allowed  to  evaluate  until  they  take  appropriate  training 
and  can  meet  them.   The  implementation  of  this 
recommendation  would  do  a  great  deal  to  improve  evaluator 
reliability  which  at  present  is  suspect  in  many  cases 
based  upon  the  data  we  have  reviewed.   It  would  also 
greatly  improve  the  feedback,  conferencing,  and 
professional  development  planning  processes  which  have 
received  criticism  from  teachers. 

If  consensus  among  evaluators  is  to  be  widely  used  as 
a  procedure  for  arriving  at  TPAI  ratings  and  identifying 
teacher  strengths  and  needs  for  improvement  as  we 
recommend  elsewhere,  an  additional  type  of  training  for 
evaluators  (all  groups)  will  be  needed — training  in 
achieving  data-based  consensus.   It  is  very  easy  in  a 
consensus  process  for  one  person's  personality  to  dominate 
and  for  ultimate  decisions  to  be  based  on  feelings  or 
impressions  rather  than  the  data  collected  by  the 
consensus  participants.   Several  models  for  consensus 
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training  for  evaluation  purposes  already  exist.   At  least 
one  nearby  program  which  has  been  reasonably  successful  is 
that  developed  by  French,  Malo  and  Chalky  for  the 
Tennessee  Career  Ladder  Program. 
Use  for  Summative  Purposes 

Is  the  use  of  the  TPAI  sufficiently  well  coordinated 
and  conducted  with  adequate  consistency  from  one  time  and 
place  to  another  to  serve  the  summative  purposes  for  which 
It  is,  in  part,  intended? 

The  system  it  is  designed  to  serve  is  statewide  in  its 
purposes,  i.e.  teacher  certification  and  career  ladder 
placement.   It  is  part  of  the  total  reform  effort  of  the 
legislature.   Many  of  the  main  features  are  actually 
prescribed  in  the  legislation  itself.   Of  necessity,  then, 
much  of  the  major  direction  and  responsibility  for  running 
this  program  does  and  probably  must  come  from  Raleigh  and 
the  DPI  in  particular. 

The  system  for  which  the  instruments  are  being  used  is 
still  evolving,  with  career  ladder  II  decisions  having 
been  made  for  the  first  time  only  this  past  year;  career 
level  III  criteria  and  accompanying  instrumentation,  have 
not  yet  been  prescribed.   Even  in  the  pilot  districts,  the 
tempo  of  career  development  activity  has  been  stepped  up 
considerably  over  what  it  will  eventually  be,  if  the 
program  is  permanently  installed.   We  recognize  that  all 
procedures  are  not  yet  fully  developed  or  operationally 
consistent  across  and  between  districts.   The  various  DPI 
reports,  newsletters  and  memoranda,  and  coordinator  and 

34 


steering  committee  meetings  are  serving  important 
coordinating  and  continuing  development  functions  as  the 
system  evolves.   The  DPI  staff,  along  with  district 
administrators,  coordinators,  and  evaluators  are  to  be 
commended  for  the  several  high  level  efforts  being  made  to 
continue  to  study,  develop,  and  refine  procedures. 

Recognizing  considerable  effort  to  coordinate  all  this 
activity,  we  also  note  a  considerable  amount  of 
inconsistency  across  and  between  the  pilot  districts  in 
how  the  instrument  is  used  to  gather  and  analyze  data  and 
in  how  final  summative  ratings  are  made.   Disparities  are 
evident  from  one  district  to  another,  and  one  school  to 
another,  in  the  percentages  of  teachers  at  or  below 
standard  performance  as  well  as  percentages  of  teachers 
qualifying  for  career  level  I  and  career  level  II.   While 
such  differences  may  reflect  real  differences  in  the 
overall  quality  of  teaching  performance  from  one  place  to 
another,  we  suspect  they  probably  indicate  differences 
from  one  evaluator  to  another  in  a.)  the  interpretation  of 
scale  descriptors,  b.)  the  use  of  recorded  information, 
and  c.)  the  sources  of  data  being  considered.   This  latter 
is  particularly  likely  for  current  functions  6,  7,  and  8 
which  are  less  observable  in  the  classroom.   We  believe 
there  is  a  direct  correlation  between  the  amount  and 
quality  of  the  training  of  evaluators  and  the  quality  of 
their  evaluations. 
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Substantial  differences  are  noted  also  in  a.)  who  is 
involved  in  achieving  sununative  ratings,  b.)  how  observer 
evaluators  perform  their  functions,  c.)  when  teachers  are 
observed  during  the  year,  and  d.)  how  the  appeals  process 
functions. 

The  amount  of  variation  in  all  these  procedures,  while 
substantial  in  some  instances,  is  understandable  given  the 
early  developmental  stage  of  the  program.   Furthermore, 
the  coordination  efforts  are  not  only  tending  to  reduce 
the  discrepancies  where  they  exist  (e.g.  most  districts 
now  require  consensus  judgment  by  OEs  and  principals  in 
making  summative  ratings)  but  there  is  healthy  development 
of  specific  operational  procedures  at  the  local  district 
levels  which  tend  to  feed  back  into  the  overall  plan  and 
help  refine  the  state  system.   An  appropriate  balance 
between  state  and  local  direction  is  important  to  the 
success  of  such  an  effort. 

To  improve  the  procedures  for  collecting  and  analyzing 
information  for  making  fair  and  equitable  personnel 
decisions,  we  urge  consideration  of  the  following 
recommendations : 

1.    Restrict  the  decision-making  use  of  the 

instrument  to  determination  of  continuing 
certification,  career  ladder  I  and  career  ladder 
II  decisions.   If  career  ladder  III  criteria, 
when  developed,  should  require  a  distinction  be 
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made  on  the  quality  of  teaching  between  level  II 
and  level  III  teachers,  we  do  not  believe  this 
instrument,  even  with  refinements,  would  be 
adequate.   Other  sources  of  data  would  be  needed. 
If  level  III  differs  from  level  II  primarily 
because  of  other  criteria,  then  other 
instrumentation  will  be  needed  also. 

2.  Intensify  training  requirements  for  all 
evaluators  including  frequent  tests  of 
evaluators'  ability  to  use  the  instruments 
accurately.   All  individuals  who  participate  as 
evaluators  should  be  certified  for  that  role  by 
passing  a  performance-based  test  of  their  ability 
to  evaluate  accurately  against  a  standard. 
Holding  such  a  certificate  should  be  prerequisite 
to  serving  as  a  summative  evaluator. 

3.  Require  joint  participation  of  OEs,  principals 
and  other  evaluators  who  are  involved  in  the  data 
collection  process  in  the  final  summative 
judgment.   Most  of  the  districts  already  seem  to 
be  doing  this.   The  use  of  multiple  judges 
strengthens  the  validity  of  the  rating  process. 
Also  OEs  as  a  group  are  the  most  highly  skilled 
evaluators  since  this  is  their  primary  function. 
Their  judgment  should  be  at  least  as  great  an 
influence  on  the  final  decision  as  that  of  the 
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principals. 

Experiment  next  year  with  observing  teachers 
three  or  four  days  in  a  row  rather  than  their 
teaching  of  single  lessons  at  different  times  of 
the  year.   We  believe  that  a  more  complete 
description  and  analysis  of  total  teaching 
performance  might  be  gained  by  seeing  how 
instruction  is  conducted  on  consecutive  days. 
The  continuity  of  instruction  from  one  day  to 
another  could  be  seen  as  well  as  how  reteaching 
is  handled  for  those  pupils  who  fail  to  learn  on 
a  first  occasion.   Teachers  would  have  greater 
opportunity  than  at  present  to  demonstrate  a 
variety  of  lesson  formats  and  show  how  creative 
and  flexible  they  can  be  in  adapting  generic 
skills  to  a  variety  of  ways  and  settings.   If  the 
career  ladder  becomes  permanently  installed,  one 
requirement  for  level  II  might  be  to  be  observed 
teaching  the  same  class  for  four  consecutive  days 
during  at  least  one  of  the  two  years  before 
promotion.   We  believe  this  would  provide  a  more 
rigorous  and  valid  performance  test  than  the 
current  schedule  of  unrelated  single  lessons. 
Create  panels  of  administrators  and  teachers  to 
review  and  rate  examples  of  FODIs  and  FODAs  for 
comparability.   Encourage  increasing  exchange  of 
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principals  and  other  evaluators  across  school  and 
district  lines  to  conduct  joint  observations, 
gather  data,  and  share  analyses.   Some  districts 
have  been  doing  this  on  an  informal  basis.   The 
more  that  is  done,  the  more  likely  it  is  that 
consistency  of  instrument  interpretation  and  use 
will  increase. 
6.   Encourage  research  study  of  rating  reliability 
and  validity  in  local  districts  as  well  as 
statewide  as  important  formative  activities  for 
evaluators.   OEs  and  principal  evaluators  should 
be  independently  reading  and  rating  the  same 
data.   Comparisons  should  be  made  between  data 
collected  at  one  time  on  a  teacher  and  that 
collected  at  another  to  see  how  much  stability 
there  is  in  ratings  of  the  same  persons  over  time 
and  setting. 

Validity  studies  of  two  types  are  needed: 
(a)  Achievement  gain  scores  of  students  taught  by 
teachers  rated  "at  standard"  should  be  compared 
with  those  of  students  taught  by  teachers  rated  5 
and  6.   (b)  Through  studies  of  the  state  test 
data  base,  student  gains  in  the  pilot  districts 
should  be  compared  with  those  in  comparable  non- 
pilot  districts. 
7.   To  provide  greater  research  capability,  we  urge 
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involvement  of  university  faculty  and  graduate 

students  in  the  study  of  evaluator  consistency 

and  practice  in  the  pilot  effort.   There  is  a 

wealth  of  information  available  for  conducting 

such  studies  and  too  few  resources  in-house  to 

get  it  done.   Modest  stipends  for  graduate 

dissertation  research  could  provide  a  wealth  of 

useful,  valuable  information  regarding  the 

system. 

8.    Provide  evaluators  with  word  processors  and 

appropriate  training  as  needed  to  assist  them  in 

writing  FODIs  and  FODAs.   The  amount  of  time 

spent  rewriting  FODIs  and  preparing  FODAs  could 

be  reduced  substantially  if  evaluators  had  such 

equipment  and  were  well  trained  in  its  use.   The 

quality  of  these  records  should  improve  as  well. 

Use  for  Formative  Purposes 

Are  the  instruments  being  used  effectively  in 
formative  ways  to  assist  teachers  and  improve  instruction? 

Formative  assistance  of  a  formal  nature  that  is  tied 

to  the  TPAI  typically  takes  the  form  of  OE-led  workshops 

for  teachers  and  other  staff  development  activities 

focused  on  one  or  more  of  the  functions.   Those  teachers 

who  have  participated  in  these  workshops  are  typically 

less  anxious  and  more  supportive  of  the  evaluation  process 

than  other  teachers.   The  more  valuable  activities  seem  to 

be  those  that  are  well  planned  and  conducted  and  involve 
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participants  in  performing  observations  of  actual  teaching 
or  videotaped  lessons. 

Regarding  individual  formative  help,  we  heard  or  saw 
many  teacher  reports  which  indicated  that  the  use  of  the 
TPAI  had  sharpened  skills  and  improved  teaching.   The 
emphasis  on  specific  teaching  functions  and  the 
specificity  of  feedback,  while  often  stressful,  was  also 
quite  helpful.   One  critical  problem,  however,  was  the 
need  for  much  greater  follow-up  assistance  by  OEs, 
especially  for  teachers  demonstrating  skill  deficiency. 
Evaluators — both  OEs  and  principals — were  just  too  busy, 
to  provide  the  extra  help  needed.   When  a  teacher  is  not 
up  to  standard  or  is  having  difficulty  improving 
performance,  OEs  typically  notify  principals  of  the 
presence  of  difficulties  but  have  little  time  themselves 
to  provide  individualized  help  and  follow-up  assistance. 

There  is  ample  evidence  that  the  current  ratio  of  96 
teachers  per  observer-evaluator  is  too  high  if  the 
observer-evaluators  are  to  provide  the  follow-through 
necessary  to  ensure  professional  improvement.   At  an 
average  of  two  observations  a  day,  with  related  FODI, 
FODA,  and  conference  obligations,  OEs  do  not  have  enough 
time  to  conduct  two  observations  of  each  teacher  much  less 
provide  follow-up  assistance.   We  recommend  a  maximum 
caseload  per  observer-evaluator  of  48  teachers. 

Increasing  the  number  of  OEs  would  also  permit  greater 
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differentiation  of  evaluation  assignments  than  is  now 
possible  to  take  advantage  of  their  teaching 
specializations  by  grade-level  and  subject  matter.   Such 
differentiation  would  help  in  evaluating  our  proposed 
content  function  in  the  TPAI . 

Attention  also  needs  to  be  directed  in  some  districts 
toward  specific  instructional  strategies  and  resources 
available  to  improve  performance  on  specific  functions. 
Other  types  of  observational  data  such  as  checklists  and 
low  inference  coding  systems  (see  Good  and  Brophy,  1987; 
McNergney  and  Carrier,  1981;  Brandt,  1981)  could  be  used 
to  highlight  differences  in  lesson  videotapes  on  such 
variables  as  time  on  tasks,  reinforcement  patterns,  and 
student  participation  rates. 

Professional  development  is  not  a  simple  phenomenon. 
Teachers  grow  in  at  least  three  different  ways.   They  grow 
toward  competence,  they  amplify  on  their  strengths  (areas 
of  competence) ,  and  they  grow  beyond  competence, 
developing  unique  talents  and  perspectives.   The  TPAI, 
PPAI,  and  PDP  are  suited  to  the  first  two  types  of 
professional  development,  but  probably  not  the  third. 

Why,  for  example,  is  the  TPAI  unlikely  to  promote 
growth  beyond  competence?   The  TPAI  currently  is  based  on 
a  set  of  essential  functions  that  must  be  mastered  by  all 
competent  teachers.   These  functions  acknowledge  the 
similarities  that  cut  across  teachers  of  all  subjects  and 
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age-groups.   Growth  beyond  competence,  however,  must  take 
account  of  each  teacher's  unique  circumstances,  including 
his/her  prior  experiences,  strengths,  subject  matter  area, 
and  students.   The  TPAI  is  not  well-equipped  to  handle 
these  areas  of  individual  difference. 

In  the  context  of  the  Career  Development  Program,  the 
TPAI,  PPAI  and  PDP  seem  to  be  most  appropriate  for 
decisions  regarding  career  levels  I  and  II.   To  the  extent 
that  career  level  III  involves  growth  beyond  competence, 
it  is  unlikely  to  be  well-served  by  these  instruments. 

While  it  was  not  our  charge  to  comment  on  career  level 
III,  we  feel  that  four  alternative  strategies  could  be 
employed. 

First,  the  same  set  of  teaching  functions  that 
constitute  the  TPAI  (or  PPAI)  could  be  used,  but  the  level 
of  expected  performance  for  each  function  could  be  raised 
substantially  above  career  level  II  expectations.   It  is 
our  judgment,  however,  that  it  would  be  difficult  to 
operationalize  and  observe  substantially  greater 
performance  in  many  of  the  existing  functions.   There  are 
practical  limits,  for  instance,  to  on-task  behavior  and 
effective  classroom  management. 

A  second  strategy  would  be  to  develop  a  set  of  master 
teacher  functions  for  career  level  III  that  totally  differ 
from  those  used  in  the  TPAI.   These  functions  might  look  a 
lot  like  the  leadership  functions  that  now  constitute  the 
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PPAI. 

A  third  strategy  would  permit  each  aspiring  career 
level  III  teacher  to  negotiate  unique  professional 
development  goals,  based  on  such  factors  as  their 
assignment,  experience,  students,  and  local  setting. 
Accomplishment  of  these  goals,  based  on  external  review 
and  assessment,  would  result  in  movement  from  career  level 
II  to  career  level  III.   The  Professional  Development 
Plan,  if  it  could  be  de-coupled  from  the  current  or 
proposed  set  of  basic  teaching  functions,  might  be  well- 
suited  to  this  third  strategy. 

A  fourth  strategy  would  involve  some  combination  of 
the  preceding  strategies. 

A  critical  question  that  must  be  addressed  before  a 
plan  for  career  level  III  can  be  developed  is  "To  what 
extent  must  all  candidates  for  career  level  III  complete 
identical  or  comparable  activities?"   Our  feeling  is  that 
the  more  veteran  teachers  are  required  to  conform  to 
similar  standards,  the  less  likely  they  will  be  to  engage 
in  meaningful  professional  development  beyond  competence. 
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APPRAISAL  OF  PRINCIPALS  AND  OTHERS 

In  addition  to  the  TPAI ,  the  review  panel  was  asked  to 
evaluate  the  PPAI .   Our  assessment  in  this  regard  is  much 
more  tentative,  given  the  lack  of  supporting  documentation 
and  field  data  and  the  early  stage  of  the  instrument's 
development.   We  feel  that  our  reactions  can  best  be 
expressed  as  a  series  of  questions: 

1.  For  what  purposes  is  the  PPAI  intended? 

We  assume  that  the  PPAI  and  TPAI  are  intended  to  serve  the 
same  purposes. 

2.  Why  are  the  performance  functions  for  the  PPAI 
more  comprehensive  than  those  for  the  TPAI? 

The  functions  for  the  TPAI  are  not  intended  to  encompass 
all  dimensions  of  the  teacher's  job.   For  example,  the 
teacher's  handling  of  curriculum  content  and  affective 
characteristics  were  not  covered  originally.   In  the  case 
of  the  PPAI,  however,  an  effort  apparently  was  made  to 
include  all  functions  of  the  principal.   This  fact  may 
have  implications  for  PPAI-based  decisions  involving 
principals  below  career  level  II. 

3.  Upon  what  data  sources  are  those  using  the  PPAI 
to  rely? 

We  could  find  few  guidelines  or  suggestions  concerning 
data  sources  to  be  tapped  by  those  completing  the  PPAI. 
Is  it  admissible,  for  example,  to  utilize  student 
achievement  data  and  teacher  input?   What  role  are 
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observations  of  principals  to  play?   Are  the  FODI  and  FODA 
to  be  employed  following  observations? 

The  review  panel  feels  that  the  list  of  PPAI  functions 
is  reasonably  well -documented  and  comprehensive.   The  PPAI 
functions  are  representative  of  our  current  knowledge  of 
the  job  expectations  of  principals,  including 
instructional  leadership.   Our  uncertainty  rests  with  how 
the  PPAI  is  to  be  used  for  particular  purposes. 

One  further  issue  must  be  noted.   If  the  principal's 
role  in  teacher  evaluation  is  crucial  enough  to  merit  the 
attention  it  has  received  in  North  Carolina  and  elsewhere, 
it  makes  sense  to  accord  the  teacher  evaluation  function 
in  the  PPAI  greater  weight  than  other  functions. 
Principals  invariably  have  more  to  do  than  time  available 
(Duke,  1987) .   By  stressing  the  importance  of  teacher 
evaluation  through  the  principal  evaluation  process,  those 
who  supervise  principals  will  be  able  to  ensure  that  the 
intent  of  the  TPAS  will  be  realized. 

Regarding  the  evaluation  of  other  certificated 
personnel,  the  review  panel  feels  uneasy  about  making 
evaluative  statements  about  the  job  descriptions  and 
performance  appraisal  instruments  for  the  following 
personnel: 

o    Assistant  Principals 

o    School  Counselors 

o    Media  Coordinators 
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o    School  Psychologists 

o    School  Social  Workers 

o    Speech-Language  Specialists 

o    Coordinators 

o    Directors 

o    General  Supervisors 

o    Observer/Evaluators 

Our  uneasiness  is  based  on  lack  of  technical  expertise 
in  r.any  of  the  areas  listed  above  and  lack  of  supporting 
materials  and  field  data  on  the  use  of  appraisal 
instruments  for  these  roles.   We  can  offer,  however,  a  few 
general  observations — 

o    Parallel  structure; 

We  are  unclear  why  certain  functions,  such  as 
"keeps  own  professional  competence  current,"  are 
used  for  certain  roles,  but  not  others. 
Presumably,  functions  of  this  kind  are 
appropriate  for  all  roles.   In  addition,  there  is 
a  need  to  make  uniform  the  wording  of  similar 
functions.   For  example,  assistant  principals  are 
expected  to  "keep  own  professional  competence 
current,"  counselors  to  "take  part  in 
professional  development  activities  to  improve 
knowledge  and  skills,"  and  general  supervisors  to 
"upgrade  own  professional  competence." 
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Technical  Expertise; 

It  is  not  clear  whether  the  persons  evaluating 
these  personnel  always  possess  expertise  in  the 
particular  area  of  responsibility.   Will 
provisions  be  made,  particularly  in  the  case  of 
smaller  units,  for  training  and/or  use  of  outside 
evaluators? 
Selective  evaluation; 

In  the  guidelines  for  the  evaluation  of  school 
psychologists,  counselors,  and  social  workers, 
provision  is  made  for  supervisor/ evaluators  and 
the  individual  to  be  evaluated  to  "mutually 
define  the  job  functions  and  competencies  to  be 
evaluated  at  the  beginning  of  each  year." 
Furthermore,  functions  and/or  competencies  "may 
be  modified  to  meet  the  specified  role  of  an 
individual.  .  .  in  the  local  education  agency." 
We  wonder  why  similar  latitude  and  discretion  are 
not  accorded  other  personnel,  including  teachers 
and  principals. 
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A  FINAL  RECOMMENDATION 

If  the  recomnendations  we  have  made  are  to  be  acted  on 
this  summer,  the  Department  of  Public  Instruction  will 
need  two  kinds  of  extra  assistance:   (1)  Additional 
personnel  should  be  hired  for  the  summer  months  to  rewrite 
materials,  forms  and  instruments.   Some  of  the  district 
leadership  familiar  with  the  TPAS,  i.e.  OEs  and 
coordinators  especially,  ought  to  be  very  capable  for  this 
task.   (2)  Experts  in  structured  interview  and  portfolio 
design  ought  to  be  added  to  the  staff  temporarily  to 
provide  technical  assistance  on  the  new  instrumentation. 

The  primary  reason  for  this  addition  of  personnel  is 
the  heavy  work  load  of  the  DPI  staff  responsible  for  this 
program.   They  are  already  working  overtime  providing 
coordinated  direction  of  the  program.   The  changes  we  are 
recommending,  while  not  altering  the  fundamental  character 
of  the  instruments,  mean  a  great  deal  of  extra  work  in  the 
short  term  to  accomplish  a  smooth  transformation  to  the 
new  materials  and  directives  that  will  be  needed. 
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